{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yfv52r2G33jY"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/AI4Finance-Foundation/FinRL/blob/master/Stock_NeurIPS2018.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gXaoZs2lh1hi"
   },
   "source": [
    "# Deep Reinforcement Learning for Stock Trading from Scratch: Multiple Stock Trading\n",
    "\n",
    "* **Pytorch Version** \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lGunVt8oLCVS"
   },
   "source": [
    "# Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HOzAKQ-SLGX6"
   },
   "source": [
    "* [1. Task Description](#0)\n",
    "* [2. Install Python packages](#1)\n",
    "    * [2.1. Install Packages](#1.1)    \n",
    "    * [2.2. A List of Python Packages](#1.2)\n",
    "    * [2.3. Import Packages](#1.3)\n",
    "    * [2.4. Create Folders](#1.4)\n",
    "* [3. Download and Preprocess Data](#2)\n",
    "* [4. Preprocess Data](#3)        \n",
    "    * [4.1. Technical Indicators](#3.1)\n",
    "    * [4.2. Perform Feature Engineering](#3.2)\n",
    "* [5. Build Market Environment in OpenAI Gym-style](#4)  \n",
    "    * [5.1. Data Split](#4.1)  \n",
    "    * [5.3. Environment for Training](#4.2)    \n",
    "* [6. Train DRL Agents](#5)\n",
    "* [7. Backtesting Performance](#6)  \n",
    "    * [7.1. BackTestStats](#6.1)\n",
    "    * [7.2. BackTestPlot](#6.2)   \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sApkDlD9LIZv"
   },
   "source": [
    "<a id='0'></a>\n",
    "# Part 1. Task Discription"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HjLD2TZSLKZ-"
   },
   "source": [
    "We train a DRL agent for stock trading. This task is modeled as a Markov Decision Process (MDP), and the objective function is maximizing (expected) cumulative return.\n",
    "\n",
    "We specify the state-action-reward as follows:\n",
    "\n",
    "* **State s**: The state space represents an agent's perception of the market environment. Just like a human trader analyzing various information, here our agent passively observes many features and learns by interacting with the market environment (usually by replaying historical data).\n",
    "\n",
    "* **Action a**: The action space includes allowed actions that an agent can take at each state. For example, a ∈ {−1, 0, 1}, where −1, 0, 1 represent\n",
    "selling, holding, and buying. When an action operates multiple shares, a ∈{−k, ..., −1, 0, 1, ..., k}, e.g.. \"Buy\n",
    "10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or −10, respectively\n",
    "\n",
    "* **Reward function r(s, a, s′)**: Reward is an incentive for an agent to learn a better policy. For example, it can be the change of the portfolio value when taking a at state s and arriving at new state s',  i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio values at state s′ and s, respectively\n",
    "\n",
    "\n",
    "**Market environment**: 30 consituent stocks of Dow Jones Industrial Average (DJIA) index. Accessed at the starting date of the testing period.\n",
    "\n",
    "\n",
    "The data for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ffsre789LY08"
   },
   "source": [
    "<a id='1'></a>\n",
    "# Part 2. Install Python Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uy5_PTmOh1hj"
   },
   "source": [
    "<a id='1.1'></a>\n",
    "## 2.1. Install packages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mPT0ipYE28wL",
    "outputId": "02a8a804-d120-4388-a167-20a81cb33d87"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting git+https://github.com/AI4Finance-Foundation/FinRL.git\n",
      "  Cloning https://github.com/AI4Finance-Foundation/FinRL.git to c:\\users\\weijia~1\\appdata\\local\\temp\\pip-req-build-ij34gjie\n",
      "  Resolved https://github.com/AI4Finance-Foundation/FinRL.git to commit 6c3a431b98db38157e179596e12a44d76a8b8922\n",
      "  Installing build dependencies: started\n",
      "  Installing build dependencies: finished with status 'done'\n",
      "  Getting requirements to build wheel: started\n",
      "  Getting requirements to build wheel: finished with status 'done'\n",
      "  Preparing metadata (pyproject.toml): started\n",
      "  Preparing metadata (pyproject.toml): finished with status 'done'\n",
      "Collecting elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl\n",
      "  Cloning https://github.com/AI4Finance-Foundation/ElegantRL.git to c:\\users\\weijia~1\\appdata\\local\\temp\\pip-install-0brexoaz\\elegantrl_909f59782e3c40bb925ed07302d3be3d\n",
      "  Resolved https://github.com/AI4Finance-Foundation/ElegantRL.git to commit f0f518e8412eca961d83c60f80c54f6d45946d8f\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2\n",
      "  Cloning https://github.com/quantopian/pyfolio.git to c:\\users\\weijia~1\\appdata\\local\\temp\\pip-install-0brexoaz\\pyfolio_ec6b2e39ef5f4d81a1b0fc3372a029e6\n",
      "  Resolved https://github.com/quantopian/pyfolio.git to commit 4b901f6d73aa02ceb6d04b7d83502e5c6f2e81aa\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting alpaca_trade_api>=2.1.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/79/25/af4a06a9241d2ce7dc6870e0b0ddac0dfd07decab3ec022855eb6b90d17d/alpaca_trade_api-2.3.0-py3-none-any.whl (33 kB)\n",
      "Requirement already satisfied: importlib-metadata==4.13.0 in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (4.13.0)\n",
      "Requirement already satisfied: matplotlib in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (3.2.2)\n",
      "Collecting exchange_calendars==3.6.3\n",
      "  Using cached exchange_calendars-3.6.3-py3-none-any.whl\n",
      "Collecting tensorboardX\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/96/47/9004f6b182920e921b6937a345019c9317fda4cbfcbeeb2af618b3b7a53e/tensorboardX-2.5.1-py2.py3-none-any.whl (125 kB)\n",
      "Requirement already satisfied: yfinance in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (0.1.87)\n",
      "Requirement already satisfied: scikit-learn>=0.21.0 in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (1.0.2)\n",
      "Collecting ray[default,tune]==1.3.0\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/3c/f280f79f6016685da8587dea2206fde47f5fba96f79a2ddde3516373678f/ray-1.3.0-cp37-cp37m-win_amd64.whl (15.6 MB)\n",
      "Collecting gputil\n",
      "  Using cached GPUtil-1.4.0-py3-none-any.whl\n",
      "Requirement already satisfied: stable-baselines3<2.0.0,>=1.6.2 in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (1.6.2)\n",
      "Collecting jqdatasdk\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/40/1c/1208b05b0d2e714bd3d46b6384f4cdc4fe0cea2f62e1d923f68247304921/jqdatasdk-1.8.11-py3-none-any.whl (158 kB)\n",
      "Collecting ccxt>=1.66.32\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/00/86/d64835dae1d03f2e511a1f11402e568141d276d34bc86802efa25c8078d8/ccxt-2.2.40-py2.py3-none-any.whl (3.5 MB)\n",
      "Collecting lz4\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9a/da/9f53e816ae10629f3b8f396d8448e7a8ba74d6ca2b5a6695b175dd559a76/lz4-4.0.2-cp37-cp37m-win_amd64.whl (98 kB)\n",
      "Requirement already satisfied: numpy>=1.17.3 in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (1.21.6)\n",
      "Collecting stockstats>=0.4.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/de/73/9899b74aee28bc657d86477ec4d8d741cf7d7b1a50ee5ebd427fe44db03f/stockstats-0.5.1-py2.py3-none-any.whl (20 kB)\n",
      "Requirement already satisfied: pandas>=1.1.5 in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (1.3.5)\n",
      "Requirement already satisfied: gym>=0.17 in d:\\ailabx\\venv\\lib\\site-packages (from finrl==0.3.5) (0.21.0)\n",
      "Requirement already satisfied: pytz in d:\\ailabx\\venv\\lib\\site-packages (from exchange_calendars==3.6.3->finrl==0.3.5) (2022.6)\n",
      "Requirement already satisfied: python-dateutil in d:\\ailabx\\venv\\lib\\site-packages (from exchange_calendars==3.6.3->finrl==0.3.5) (2.8.2)\n",
      "Collecting korean-lunar-calendar\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9c/96/30f3fe51b336bb6da4714f4fdad7bbdce8f13af79af2eb75e22908f3f9f4/korean_lunar_calendar-0.3.1-py3-none-any.whl (9.0 kB)\n",
      "Collecting toolz\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7f/5c/922a3508f5bda2892be3df86c74f9cf1e01217c2b1f8a0ac4841d903e3e9/toolz-0.12.0-py3-none-any.whl (55 kB)\n",
      "Collecting pyluach\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b9/8b/29d15f7b4b9f9487ae9f4754142ddc496708ad8177b0a54abeab3f638f9e/pyluach-2.0.2-py3-none-any.whl (22 kB)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in d:\\ailabx\\venv\\lib\\site-packages (from importlib-metadata==4.13.0->finrl==0.3.5) (4.4.0)\n",
      "Requirement already satisfied: zipp>=0.5 in d:\\ailabx\\venv\\lib\\site-packages (from importlib-metadata==4.13.0->finrl==0.3.5) (3.11.0)\n",
      "Collecting aioredis\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9b/a9/0da089c3ae7a31cbcd2dcf0214f6f571e1295d292b6139e2bac68ec081d0/aioredis-2.0.1-py3-none-any.whl (71 kB)\n",
      "Requirement already satisfied: grpcio>=1.28.1 in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (1.50.0)\n",
      "Collecting aiohttp-cors\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/13/e7/e436a0c0eb5127d8b491a9b83ecd2391c6ff7dcd5548dfaec2080a2340fd/aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)\n",
      "Collecting aiohttp\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/6a/db/7dd37ac95945fd259b3b6c9ff1a3944129f181a6a1912dcf473df3f505e2/aiohttp-3.8.3-cp37-cp37m-win_amd64.whl (322 kB)\n",
      "Collecting redis>=3.5.0\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ad/dd7b6423295394b95e03d961d454e3046b569715dcc2dd4a030bb43a7cff/redis-4.3.5-py3-none-any.whl (248 kB)\n",
      "Requirement already satisfied: requests in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (2.28.1)\n",
      "Collecting filelock\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/94/b3/ff2845971788613e646e667043fdb5f128e2e540aefa09a3c55be8290d6d/filelock-3.8.0-py3-none-any.whl (10 kB)\n",
      "Collecting opencensus\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f1/1a/241693f49e696bc895b0a51574284e676193c92efda6e5fc89036898ea68/opencensus-0.11.0-py2.py3-none-any.whl (128 kB)\n",
      "Requirement already satisfied: prometheus-client>=0.7.1 in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (0.15.0)\n",
      "Collecting py-spy>=0.2.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9d/eb/1749a892ada87c65320503d8be2aa31ba1e034cc818aead54ec45d347719/py_spy-0.3.14-py2.py3-none-win_amd64.whl (1.4 MB)\n",
      "Collecting msgpack<2.0.0,>=1.0.0\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/05/46/633f2a31bf0e6e5530993bc8c8230d165323bd372eb50b8c9f74ca04fc68/msgpack-1.0.4-cp37-cp37m-win_amd64.whl (61 kB)\n",
      "Requirement already satisfied: protobuf>=3.15.3 in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (3.19.6)\n",
      "Requirement already satisfied: pyyaml in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (6.0)\n",
      "Collecting gpustat\n",
      "  Using cached gpustat-1.0.0-py3-none-any.whl\n",
      "Requirement already satisfied: colorama in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (0.4.6)\n",
      "Collecting click>=7.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c2/f1/df59e28c642d583f7dacffb1e0965d0e00b218e0186d7858ac5233dce840/click-8.1.3-py3-none-any.whl (96 kB)\n",
      "Requirement already satisfied: jsonschema in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (4.17.1)\n",
      "Collecting colorful\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b0/8e/e386e248266952d24d73ed734c2f5513f34d9557032618c8910e605dfaf6/colorful-0.5.4-py2.py3-none-any.whl (201 kB)\n",
      "Requirement already satisfied: tabulate in d:\\ailabx\\venv\\lib\\site-packages (from ray[default,tune]==1.3.0->finrl==0.3.5) (0.9.0)\n",
      "Requirement already satisfied: websocket-client<2,>=0.56.0 in d:\\ailabx\\venv\\lib\\site-packages (from alpaca_trade_api>=2.1.0->finrl==0.3.5) (1.4.2)\n",
      "Collecting deprecation==2.1.0\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/02/c3/253a89ee03fc9b9682f1541728eb66db7db22148cd94f89ab22528cd1e1b/deprecation-2.1.0-py2.py3-none-any.whl (11 kB)\n",
      "Requirement already satisfied: urllib3<2,>1.24 in d:\\ailabx\\venv\\lib\\site-packages (from alpaca_trade_api>=2.1.0->finrl==0.3.5) (1.26.13)\n",
      "Collecting websockets<11,>=9.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ab/41/ed2fecb228c1f25cea03fce4a22a86f7771a10875d5762e777e943bb7d68/websockets-10.4-cp37-cp37m-win_amd64.whl (101 kB)\n",
      "Collecting msgpack<2.0.0,>=1.0.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/72/d3/2669479923d8e7129bb7029b5a32c19d1bb99153793e24f90e2b48ec5d5f/msgpack-1.0.3-cp37-cp37m-win_amd64.whl (68 kB)\n",
      "Collecting aiohttp\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/19/22/7a9ea2992cf7adf534eb51f7b6bfd4642bb15894ac36df4edb0838ee1fdb/aiohttp-3.8.1-cp37-cp37m-win_amd64.whl (551 kB)\n",
      "Collecting async-timeout<5.0,>=4.0.0a3\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d6/c1/8991e7c5385b897b8c020cdaad718c5b087a6626d1d11a23e1ea87e325a7/async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
      "Requirement already satisfied: attrs>=17.3.0 in d:\\ailabx\\venv\\lib\\site-packages (from aiohttp->ray[default,tune]==1.3.0->finrl==0.3.5) (22.1.0)\n",
      "Collecting asynctest==0.13.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e8/b6/8d17e169d577ca7678b11cd0d3ceebb0a6089a7f4a2de4b945fe4b1c86db/asynctest-0.13.0-py3-none-any.whl (26 kB)\n",
      "Collecting aiosignal>=1.1.2\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in d:\\ailabx\\venv\\lib\\site-packages (from aiohttp->ray[default,tune]==1.3.0->finrl==0.3.5) (2.1.1)\n",
      "Collecting yarl<2.0,>=1.0\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d1/4a/294168151f8bb9eb2c9ae0192b01a274f1a98056bfe08b80ce96f39bfb04/yarl-1.8.1-cp37-cp37m-win_amd64.whl (56 kB)\n",
      "Collecting frozenlist>=1.1.1\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/20/f9/5beca2597a95de184dc695bc6ff53e551e45dd59f7a885e52514b76bf48b/frozenlist-1.3.3-cp37-cp37m-win_amd64.whl (34 kB)\n",
      "Collecting multidict<7.0,>=4.5\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/71/20/5388d06fab6600b61ff2cfc0ed87a7a8a09d25be9b269ff48d413e8ddb87/multidict-6.0.2-cp37-cp37m-win_amd64.whl (27 kB)\n",
      "Requirement already satisfied: packaging in d:\\ailabx\\venv\\lib\\site-packages (from deprecation==2.1.0->alpaca_trade_api>=2.1.0->finrl==0.3.5) (21.3)\n",
      "Collecting aiodns>=1.1.1\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ab/72/991ee33a517df69c6cd6f3486cfe9b6329557cb55acaa8cefac33c2aa4d2/aiodns-3.0.0-py3-none-any.whl (5.0 kB)\n",
      "Collecting setuptools>=60.9.0\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ef/e3/29d6e1a07e8d90ace4a522d9689d03e833b67b50d1588e693eec15f26251/setuptools-65.6.3-py3-none-any.whl (1.2 MB)\n",
      "Requirement already satisfied: certifi>=2018.1.18 in d:\\ailabx\\venv\\lib\\site-packages (from ccxt>=1.66.32->finrl==0.3.5) (2022.9.24)\n",
      "Requirement already satisfied: cryptography>=2.6.1 in d:\\ailabx\\venv\\lib\\site-packages (from ccxt>=1.66.32->finrl==0.3.5) (38.0.3)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in d:\\ailabx\\venv\\lib\\site-packages (from gym>=0.17->finrl==0.3.5) (2.2.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in d:\\ailabx\\venv\\lib\\site-packages (from scikit-learn>=0.21.0->finrl==0.3.5) (3.1.0)\n",
      "Requirement already satisfied: joblib>=0.11 in d:\\ailabx\\venv\\lib\\site-packages (from scikit-learn>=0.21.0->finrl==0.3.5) (1.2.0)\n",
      "Requirement already satisfied: scipy>=1.1.0 in d:\\ailabx\\venv\\lib\\site-packages (from scikit-learn>=0.21.0->finrl==0.3.5) (1.7.3)\n",
      "Requirement already satisfied: torch>=1.11 in d:\\ailabx\\venv\\lib\\site-packages (from stable-baselines3<2.0.0,>=1.6.2->finrl==0.3.5) (1.13.0)\n",
      "Collecting pybullet\n",
      "  Using cached pybullet-3.2.5-cp37-cp37m-win_amd64.whl\n",
      "Requirement already satisfied: opencv-python in d:\\ailabx\\venv\\lib\\site-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.5) (4.6.0.66)\n",
      "Collecting box2d-py\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/98/c2/ab05b5329dc4416b5ee5530f0625a79c394a3e3c10abe0812b9345256451/box2d-py-2.3.8.tar.gz (374 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting pymysql>=0.7.6\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4f/52/a115fe175028b058df353c5a3d5290b71514a83f67078a6482cff24d6137/PyMySQL-1.0.2-py3-none-any.whl (43 kB)\n",
      "Collecting SQLAlchemy>=1.2.8\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/99/12/9f5ca8939bf002ed86982c6205e0066f0c95600d4b74a0abf471245e52ea/SQLAlchemy-1.4.44-cp37-cp37m-win_amd64.whl (1.6 MB)\n",
      "Requirement already satisfied: six in d:\\ailabx\\venv\\lib\\site-packages (from jqdatasdk->finrl==0.3.5) (1.16.0)\n",
      "Collecting thriftpy2>=0.3.9\n",
      "  Using cached thriftpy2-0.4.16-py2.py3-none-any.whl\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in d:\\ailabx\\venv\\lib\\site-packages (from matplotlib->finrl==0.3.5) (1.4.4)\n",
      "Requirement already satisfied: cycler>=0.10 in d:\\ailabx\\venv\\lib\\site-packages (from matplotlib->finrl==0.3.5) (0.11.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in d:\\ailabx\\venv\\lib\\site-packages (from matplotlib->finrl==0.3.5) (3.0.9)\n",
      "Requirement already satisfied: ipython>=3.2.3 in d:\\ailabx\\venv\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (7.34.0)\n",
      "Requirement already satisfied: seaborn>=0.7.1 in d:\\ailabx\\venv\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.12.1)\n",
      "Requirement already satisfied: empyrical>=0.5.0 in d:\\ailabx\\venv\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.5.5)\n",
      "Requirement already satisfied: lxml>=4.5.1 in d:\\ailabx\\venv\\lib\\site-packages (from yfinance->finrl==0.3.5) (4.9.1)\n",
      "Requirement already satisfied: appdirs>=1.4.4 in d:\\ailabx\\venv\\lib\\site-packages (from yfinance->finrl==0.3.5) (1.4.4)\n",
      "Requirement already satisfied: multitasking>=0.0.7 in d:\\ailabx\\venv\\lib\\site-packages (from yfinance->finrl==0.3.5) (0.0.11)\n",
      "Collecting pycares>=4.0.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/58/cd/09d09f3ccbb98fdd4f542765aaeaee99d741cc9d367941e288ff5f75b7c0/pycares-4.2.2-cp37-cp37m-win_amd64.whl (75 kB)\n",
      "Requirement already satisfied: cffi>=1.12 in d:\\ailabx\\venv\\lib\\site-packages (from cryptography>=2.6.1->ccxt>=1.66.32->finrl==0.3.5) (1.15.1)\n",
      "Requirement already satisfied: pandas-datareader>=0.2 in d:\\ailabx\\venv\\lib\\site-packages (from empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.10.0)\n",
      "Requirement already satisfied: decorator in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (5.1.1)\n",
      "Requirement already satisfied: pickleshare in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.7.5)\n",
      "Requirement already satisfied: pygments in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (2.13.0)\n",
      "Requirement already satisfied: traitlets>=4.2 in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (5.5.0)\n",
      "Requirement already satisfied: matplotlib-inline in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.1.6)\n",
      "Requirement already satisfied: backcall in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.2.0)\n",
      "Requirement already satisfied: jedi>=0.16 in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.18.2)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in d:\\ailabx\\venv\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (3.0.33)\n",
      "Requirement already satisfied: idna<4,>=2.5 in d:\\ailabx\\venv\\lib\\site-packages (from requests->ray[default,tune]==1.3.0->finrl==0.3.5) (3.4)\n",
      "Collecting greenlet!=0.4.17\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/95/a4/d89fd972a69454f0cfd67192d3fdbf82d9b9a9271c30c76ea4e0e57ad745/greenlet-2.0.1-cp37-cp37m-win_amd64.whl (191 kB)\n",
      "Collecting ply<4.0,>=3.4\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49 kB)\n",
      "Collecting nvidia-ml-py<=11.495.46,>=11.450.129\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/eb/53/7ce349858b2004eea4690fdb40023bcc7f55fb0bcf719c84892058583934/nvidia_ml_py-11.495.46-py3-none-any.whl (25 kB)\n",
      "Requirement already satisfied: psutil>=5.6.0 in d:\\ailabx\\venv\\lib\\site-packages (from gpustat->ray[default,tune]==1.3.0->finrl==0.3.5) (5.9.4)\n",
      "Collecting blessed>=1.17.1\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/94/c5/651224a1bbcb0253fecb7af6f60fa0c9ebabcb1df01ceadb8f4dfd2528cb/blessed-1.19.1-py2.py3-none-any.whl (58 kB)\n",
      "Requirement already satisfied: importlib-resources>=1.4.0 in d:\\ailabx\\venv\\lib\\site-packages (from jsonschema->ray[default,tune]==1.3.0->finrl==0.3.5) (5.10.0)\n",
      "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in d:\\ailabx\\venv\\lib\\site-packages (from jsonschema->ray[default,tune]==1.3.0->finrl==0.3.5) (1.3.10)\n",
      "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in d:\\ailabx\\venv\\lib\\site-packages (from jsonschema->ray[default,tune]==1.3.0->finrl==0.3.5) (0.19.2)\n",
      "Collecting google-api-core<3.0.0,>=1.0.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/15/fb/b641357b0f8fc540c766bc8f66048de0398d0a3e0cad7c4a08e2fbee733a/google_api_core-2.10.2-py3-none-any.whl (115 kB)\n",
      "Collecting opencensus-context>=0.1.3\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/10/68/162c97ea78c957d68ecf78a5c5041d2e25bd5562bdf5d89a6cbf7f8429bf/opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB)\n",
      "Requirement already satisfied: wcwidth>=0.1.4 in d:\\ailabx\\venv\\lib\\site-packages (from blessed>=1.17.1->gpustat->ray[default,tune]==1.3.0->finrl==0.3.5) (0.2.5)\n",
      "Collecting jinxed>=1.1.0\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/23/22/9b3481b11f32aedf1cc403f290b92bd23327d04e359808482a69b2bc3665/jinxed-1.2.0-py2.py3-none-any.whl (33 kB)\n",
      "Requirement already satisfied: pycparser in d:\\ailabx\\venv\\lib\\site-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt>=1.66.32->finrl==0.3.5) (2.21)\n",
      "Collecting googleapis-common-protos<2.0dev,>=1.56.2\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f0/2a/25d8c1ceedc5af97de37434c9c5e38ce28aaa45960aa2bd7aa215fc420c0/googleapis_common_protos-1.57.0-py2.py3-none-any.whl (217 kB)\n",
      "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in d:\\ailabx\\venv\\lib\\site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]==1.3.0->finrl==0.3.5) (2.14.1)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in d:\\ailabx\\venv\\lib\\site-packages (from jedi>=0.16->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.8.3)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in d:\\ailabx\\venv\\lib\\site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]==1.3.0->finrl==0.3.5) (0.2.8)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in d:\\ailabx\\venv\\lib\\site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]==1.3.0->finrl==0.3.5) (4.9)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in d:\\ailabx\\venv\\lib\\site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]==1.3.0->finrl==0.3.5) (5.2.0)\n",
      "Collecting ansicon\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/75/f9/f1c10e223c7b56a38109a3f2eb4e7fe9a757ea3ed3a166754fb30f65e466/ansicon-1.89.0-py2.py3-none-any.whl (63 kB)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in d:\\ailabx\\venv\\lib\\site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]==1.3.0->finrl==0.3.5) (0.4.8)\n",
      "Building wheels for collected packages: finrl, elegantrl, pyfolio, box2d-py\n",
      "  Building wheel for finrl (pyproject.toml): started\n",
      "  Building wheel for finrl (pyproject.toml): finished with status 'done'\n",
      "  Created wheel for finrl: filename=finrl-0.3.5-py3-none-any.whl size=2745754 sha256=1af50f6d586c6ccb275e30f4c078601e3fd636cff28430058c82cbfeff8cc963\n",
      "  Stored in directory: C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-ephem-wheel-cache-bazfvo36\\wheels\\57\\42\\c6\\458c5fd63ad5dc7005d30864147a61e41085745eb213f4ef46\n",
      "  Building wheel for elegantrl (setup.py): started\n",
      "  Building wheel for elegantrl (setup.py): finished with status 'done'\n",
      "  Created wheel for elegantrl: filename=elegantrl-0.3.3-py3-none-any.whl size=349545 sha256=8996f05f5393d9c7003411ad15ef69613bf4cd76d81edef47cdfd667bf11c973\n",
      "  Stored in directory: C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-ephem-wheel-cache-bazfvo36\\wheels\\99\\85\\5e\\86cb3a9f47adfca5e248295e93113e1b298d60883126d62c84\n",
      "  Building wheel for pyfolio (setup.py): started\n",
      "  Building wheel for pyfolio (setup.py): finished with status 'done'\n",
      "  Created wheel for pyfolio: filename=pyfolio-0.9.2+75.g4b901f6-py3-none-any.whl size=76274 sha256=597c17918dbeef11d9a1fb2cfb372b5c422b049f5b59abff8a1d87d469596f32\n",
      "  Stored in directory: C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-ephem-wheel-cache-bazfvo36\\wheels\\ef\\09\\e5\\2c1bf37c050d22557c080deb1be986d06424627c04aeca19b9\n",
      "  Building wheel for box2d-py (setup.py): started\n",
      "  Building wheel for box2d-py (setup.py): finished with status 'error'\n",
      "  Running setup.py clean for box2d-py\n",
      "Successfully built finrl elegantrl pyfolio\n",
      "Failed to build box2d-py\n",
      "Installing collected packages: multidict, frozenlist, ansicon, yarl, jinxed, googleapis-common-protos, asynctest, async-timeout, aiosignal, opencensus-context, nvidia-ml-py, google-api-core, blessed, aiohttp, setuptools, redis, pycares, py-spy, ply, opencensus, msgpack, greenlet, gpustat, filelock, click, aioredis, aiohttp-cors, websockets, toolz, thriftpy2, tensorboardX, SQLAlchemy, ray, pymysql, pyluach, pybullet, korean-lunar-calendar, deprecation, colorful, box2d-py, aiodns, stockstats, pyfolio, lz4, jqdatasdk, gputil, exchange-calendars, elegantrl, ccxt, alpaca-trade-api, finrl\n",
      "  Attempting uninstall: setuptools\n",
      "    Found existing installation: setuptools 60.2.0\n",
      "    Uninstalling setuptools-60.2.0:\n",
      "      Successfully uninstalled setuptools-60.2.0\n",
      "    Running setup.py install for box2d-py: started\n",
      "    Running setup.py install for box2d-py: finished with status 'error'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "  Running command git clone --filter=blob:none -q https://github.com/AI4Finance-Foundation/FinRL.git 'C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-req-build-ij34gjie'\n",
      "  Running command git clone --filter=blob:none -q https://github.com/AI4Finance-Foundation/ElegantRL.git 'C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-install-0brexoaz\\elegantrl_909f59782e3c40bb925ed07302d3be3d'\n",
      "  Running command git clone --filter=blob:none -q https://github.com/quantopian/pyfolio.git 'C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-install-0brexoaz\\pyfolio_ec6b2e39ef5f4d81a1b0fc3372a029e6'\n",
      "  ERROR: Command errored out with exit status 1:\n",
      "   command: 'D:\\ailabx\\venv\\Scripts\\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'C:\\\\Users\\\\WEIJIA~1\\\\AppData\\\\Local\\\\Temp\\\\pip-install-0brexoaz\\\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\\setup.py'\"'\"'; __file__='\"'\"'C:\\\\Users\\\\WEIJIA~1\\\\AppData\\\\Local\\\\Temp\\\\pip-install-0brexoaz\\\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\\setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' bdist_wheel -d 'C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-wheel-bkylokdo'\n",
      "       cwd: C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-install-0brexoaz\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\n",
      "  Complete output (14 lines):\n",
      "  Using setuptools (version 60.2.0).\n",
      "  running bdist_wheel\n",
      "  running build\n",
      "  running build_py\n",
      "  creating build\n",
      "  creating build\\lib.win-amd64-3.7\n",
      "  creating build\\lib.win-amd64-3.7\\Box2D\n",
      "  copying library\\Box2D\\Box2D.py -> build\\lib.win-amd64-3.7\\Box2D\n",
      "  copying library\\Box2D\\__init__.py -> build\\lib.win-amd64-3.7\\Box2D\n",
      "  creating build\\lib.win-amd64-3.7\\Box2D\\b2\n",
      "  copying library\\Box2D\\b2\\__init__.py -> build\\lib.win-amd64-3.7\\Box2D\\b2\n",
      "  running build_ext\n",
      "  swig.exe -python -c++ -IBox2D -small -O -includeall -ignoremissing -w201 -globals b2Globals -outdir library\\Box2D -keyword -w511 -D_SWIG_KWARGS -o Box2D\\Box2D_wrap.cpp Box2D\\Box2D.i\n",
      "  error: command 'swig.exe' failed: None\n",
      "  ----------------------------------------\n",
      "  ERROR: Failed building wheel for box2d-py\n",
      "WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "    WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "    ERROR: Command errored out with exit status 1:\n",
      "     command: 'D:\\ailabx\\venv\\Scripts\\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'C:\\\\Users\\\\WEIJIA~1\\\\AppData\\\\Local\\\\Temp\\\\pip-install-0brexoaz\\\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\\setup.py'\"'\"'; __file__='\"'\"'C:\\\\Users\\\\WEIJIA~1\\\\AppData\\\\Local\\\\Temp\\\\pip-install-0brexoaz\\\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\\setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' install --record 'C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-record-e7su7zax\\install-record.txt' --single-version-externally-managed --compile --install-headers 'D:\\ailabx\\venv\\include\\site\\python3.7\\box2d-py'\n",
      "         cwd: C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-install-0brexoaz\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\n",
      "    Complete output (18 lines):\n",
      "    Using setuptools (version 65.6.3).\n",
      "    running install\n",
      "    D:\\ailabx\\venv\\lib\\site-packages\\setuptools\\command\\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n",
      "      setuptools.SetuptoolsDeprecationWarning,\n",
      "    running build\n",
      "    running build_py\n",
      "    creating build\n",
      "    creating build\\lib.win-amd64-cpython-37\n",
      "    creating build\\lib.win-amd64-cpython-37\\Box2D\n",
      "    copying library\\Box2D\\Box2D.py -> build\\lib.win-amd64-cpython-37\\Box2D\n",
      "    copying library\\Box2D\\__init__.py -> build\\lib.win-amd64-cpython-37\\Box2D\n",
      "    creating build\\lib.win-amd64-cpython-37\\Box2D\\b2\n",
      "    copying library\\Box2D\\b2\\__init__.py -> build\\lib.win-amd64-cpython-37\\Box2D\\b2\n",
      "    running build_ext\n",
      "    building 'Box2D._Box2D' extension\n",
      "    swigging Box2D\\Box2D.i to Box2D\\Box2D_wrap.cpp\n",
      "    swig.exe -python -c++ -IBox2D -small -O -includeall -ignoremissing -w201 -globals b2Globals -outdir library\\Box2D -keyword -w511 -D_SWIG_KWARGS -o Box2D\\Box2D_wrap.cpp Box2D\\Box2D.i\n",
      "    error: command 'swig.exe' failed: None\n",
      "    ----------------------------------------\n",
      "ERROR: Command errored out with exit status 1: 'D:\\ailabx\\venv\\Scripts\\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'C:\\\\Users\\\\WEIJIA~1\\\\AppData\\\\Local\\\\Temp\\\\pip-install-0brexoaz\\\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\\setup.py'\"'\"'; __file__='\"'\"'C:\\\\Users\\\\WEIJIA~1\\\\AppData\\\\Local\\\\Temp\\\\pip-install-0brexoaz\\\\box2d-py_b3dd5244bd8748619451120b9c128b15\\\\setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' install --record 'C:\\Users\\WEIJIA~1\\AppData\\Local\\Temp\\pip-record-e7su7zax\\install-record.txt' --single-version-externally-managed --compile --install-headers 'D:\\ailabx\\venv\\include\\site\\python3.7\\box2d-py' Check the logs for full command output.\n",
      "WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution -rotobuf (d:\\ailabx\\venv\\lib\\site-packages)\n",
      "WARNING: You are using pip version 21.3.1; however, version 22.3.1 is available.\n",
      "You should consider upgrading via the 'D:\\ailabx\\venv\\Scripts\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "## install finrl library\n",
    "!pip install git+https://github.com/AI4Finance-Foundation/FinRL.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "osBHhVysOEzi"
   },
   "source": [
    "\n",
    "<a id='1.2'></a>\n",
    "## 2.2. A list of Python packages \n",
    "* Yahoo Finance API\n",
    "* pandas\n",
    "* numpy\n",
    "* matplotlib\n",
    "* stockstats\n",
    "* OpenAI gym\n",
    "* stable-baselines\n",
    "* tensorflow\n",
    "* pyfolio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nGv01K8Sh1hn"
   },
   "source": [
    "<a id='1.3'></a>\n",
    "## 2.3. Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "lPqeTTwoh1hn"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'alpaca_trade_api'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_20552\\4094496222.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'matplotlib'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'inline'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmeta\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpreprocessor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0myahoodownloader\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mYahooDownloader\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmeta\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpreprocessor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpreprocessors\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mFeatureEngineer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata_split\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmeta\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menv_stock_trading\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menv_stocktrading\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mStockTradingEnv\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\ailabx\\bak\\FinRL-master\\FinRL-master\\finrl\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtest\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrade\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtrade\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrain\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtrain\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\ailabx\\bak\\FinRL-master\\FinRL-master\\finrl\\trade.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0m__future__\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mannotations\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmeta\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menv_stock_trading\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menv_stock_papertrading\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mAlpacaPaperTrading\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mfinrl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtest\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\ailabx\\bak\\FinRL-master\\FinRL-master\\finrl\\meta\\env_stock_trading\\env_stock_papertrading.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtime\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0malpaca_trade_api\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mtradeapi\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      8\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mgym\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'alpaca_trade_api'"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "# matplotlib.use('Agg')\n",
    "import datetime\n",
    "\n",
    "%matplotlib inline\n",
    "from finrl.meta.preprocessor.yahoodownloader import YahooDownloader\n",
    "from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split\n",
    "from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv\n",
    "from finrl.agents.stablebaselines3.models import DRLAgent\n",
    "from stable_baselines3.common.logger import configure\n",
    "from finrl.meta.data_processor import DataProcessor\n",
    "\n",
    "from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline\n",
    "from pprint import pprint\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"../FinRL\")\n",
    "\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T2owTj985RW4"
   },
   "source": [
    "<a id='1.4'></a>\n",
    "## 2.4. Create Folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RtUc_ofKmpdy"
   },
   "outputs": [],
   "source": [
    "from finrl import config\n",
    "from finrl import config_tickers\n",
    "import os\n",
    "from finrl.main import check_and_make_directories\n",
    "from finrl.config import (\n",
    "    DATA_SAVE_DIR,\n",
    "    TRAINED_MODEL_DIR,\n",
    "    TENSORBOARD_LOG_DIR,\n",
    "    RESULTS_DIR,\n",
    "    INDICATORS,\n",
    "    TRAIN_START_DATE,\n",
    "    TRAIN_END_DATE,\n",
    "    TEST_START_DATE,\n",
    "    TEST_END_DATE,\n",
    "    TRADE_START_DATE,\n",
    "    TRADE_END_DATE,\n",
    ")\n",
    "check_and_make_directories([DATA_SAVE_DIR, TRAINED_MODEL_DIR, TENSORBOARD_LOG_DIR, RESULTS_DIR])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A289rQWMh1hq"
   },
   "source": [
    "<a id='2'></a>\n",
    "# Part 3. Download Data\n",
    "Yahoo Finance provides stock data, financial news, financial reports, etc. Yahoo Finance is free.\n",
    "* FinRL uses a class **YahooDownloader** in FinRL-Meta to fetch data via Yahoo Finance API\n",
    "* Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NPeQ7iS-LoMm"
   },
   "source": [
    "\n",
    "\n",
    "-----\n",
    "class YahooDownloader:\n",
    "    Retrieving daily stock data from\n",
    "    Yahoo Finance API\n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "        start_date : str\n",
    "            start date of the data (modified from config.py)\n",
    "        end_date : str\n",
    "            end date of the data (modified from config.py)\n",
    "        ticker_list : list\n",
    "            a list of stock tickers (modified from config.py)\n",
    "\n",
    "    Methods\n",
    "    -------\n",
    "    fetch_data()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "h3XJnvrbLp-C",
    "outputId": "3c4dda81-f617-4e9b-f88c-edb502d1500c"
   },
   "outputs": [],
   "source": [
    "# from config.py, TRAIN_START_DATE is a string\n",
    "TRAIN_START_DATE\n",
    "# from config.py, TRAIN_END_DATE is a string\n",
    "TRAIN_END_DATE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FUnY8WEfLq3C"
   },
   "outputs": [],
   "source": [
    "TRAIN_START_DATE = '2009-01-01'\n",
    "TRAIN_END_DATE = '2020-07-01'\n",
    "TRADE_START_DATE = '2020-07-01'\n",
    "TRADE_END_DATE = '2021-10-31'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yCKm4om-s9kE",
    "outputId": "24dbccc1-190c-4912-b67a-9fe8fd9afbf2"
   },
   "outputs": [],
   "source": [
    "df = YahooDownloader(start_date = TRAIN_START_DATE,\n",
    "                     end_date = TRADE_END_DATE,\n",
    "                     ticker_list = config_tickers.DOW_30_TICKER).fetch_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JzqRRTOX6aFu",
    "outputId": "85539f72-49fe-4418-9a75-5fa715cbfac9"
   },
   "outputs": [],
   "source": [
    "print(config_tickers.DOW_30_TICKER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CV3HrZHLh1hy",
    "outputId": "f1031e65-9b05-4fb4-9128-3b7babd894c5"
   },
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "4hYkeaPiICHS",
    "outputId": "95d13d1e-3c8d-441e-835e-b9c7c121fd9f"
   },
   "outputs": [],
   "source": [
    "df.sort_values(['date','tic'],ignore_index=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uqC6c40Zh1iH"
   },
   "source": [
    "# Part 4: Preprocess Data\n",
    "We need to check for missing data and do feature engineering to convert the data point into a state.\n",
    "* **Adding technical indicators**. In practical trading, various information needs to be taken into account, such as historical prices, current holding shares, technical indicators, etc. Here, we demonstrate two trend-following technical indicators: MACD and RSI.\n",
    "* **Adding turbulence index**. Risk-aversion reflects whether an investor prefers to protect the capital. It also influences one's trading strategy when facing different market volatility level. To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the turbulence index that measures extreme fluctuation of asset price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PmKP-1ii3RLS",
    "outputId": "9b9facc1-f877-4caa-b66b-a2ce0e023de8"
   },
   "outputs": [],
   "source": [
    "fe = FeatureEngineer(\n",
    "                    use_technical_indicator=True,\n",
    "                    tech_indicator_list = INDICATORS,\n",
    "                    use_vix=True,\n",
    "                    use_turbulence=True,\n",
    "                    user_defined_feature = False)\n",
    "\n",
    "processed = fe.preprocess_data(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Kixon2tR3RLT"
   },
   "outputs": [],
   "source": [
    "list_ticker = processed[\"tic\"].unique().tolist()\n",
    "list_date = list(pd.date_range(processed['date'].min(),processed['date'].max()).astype(str))\n",
    "combination = list(itertools.product(list_date,list_ticker))\n",
    "\n",
    "processed_full = pd.DataFrame(combination,columns=[\"date\",\"tic\"]).merge(processed,on=[\"date\",\"tic\"],how=\"left\")\n",
    "processed_full = processed_full[processed_full['date'].isin(processed['date'])]\n",
    "processed_full = processed_full.sort_values(['date','tic'])\n",
    "\n",
    "processed_full = processed_full.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 407
    },
    "id": "grvhGJJII3Xn",
    "outputId": "8e48188a-d58d-4d2c-dbed-fecacb74984e"
   },
   "outputs": [],
   "source": [
    "processed_full.sort_values(['date','tic'],ignore_index=True).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-QsYaY0Dh1iw"
   },
   "source": [
    "<a id='4'></a>\n",
    "# Part 5. Build A Market Environment in OpenAI Gym-style\n",
    "The training process involves observing stock price change, taking an action and reward's calculation. By interacting with the market environment, the agent will eventually derive a trading strategy that may maximize (expected) rewards.\n",
    "\n",
    "Our market environment, based on OpenAI Gym, simulates stock markets with historical market data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5TOhcryx44bb"
   },
   "source": [
    "## Data Split\n",
    "We split the data into training set and testing set as follows:\n",
    "\n",
    "Training data period: 2009-01-01 to 2020-07-01\n",
    "\n",
    "Trading data period: 2020-07-01 to 2021-10-31\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W0qaVGjLtgbI",
    "outputId": "ac9f2699-65c6-418f-cca4-f0f166973f65"
   },
   "outputs": [],
   "source": [
    "train = data_split(processed_full, TRAIN_START_DATE,TRAIN_END_DATE)\n",
    "trade = data_split(processed_full, TRADE_START_DATE,TRADE_END_DATE)\n",
    "print(len(train))\n",
    "print(len(trade))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "id": "p52zNCOhTtLR",
    "outputId": "14568787-e92d-4d9b-dbce-9e11b3fb1390"
   },
   "outputs": [],
   "source": [
    "train.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "id": "k9zU9YaTTvFq",
    "outputId": "80e64947-4c5c-428f-a98c-6b344d816783"
   },
   "outputs": [],
   "source": [
    "trade.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zYN573SOHhxG",
    "outputId": "460cd177-65d2-46f8-eb40-eaeda045c4bb"
   },
   "outputs": [],
   "source": [
    "INDICATORS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Q2zqII8rMIqn",
    "outputId": "d2e2b678-04f8-44be-96ea-3523903780eb"
   },
   "outputs": [],
   "source": [
    "stock_dimension = len(train.tic.unique())\n",
    "state_space = 1 + 2*stock_dimension + len(INDICATORS)*stock_dimension\n",
    "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AWyp84Ltto19"
   },
   "outputs": [],
   "source": [
    "buy_cost_list = sell_cost_list = [0.001] * stock_dimension\n",
    "num_stock_shares = [0] * stock_dimension\n",
    "\n",
    "env_kwargs = {\n",
    "    \"hmax\": 100,\n",
    "    \"initial_amount\": 1000000,\n",
    "    \"num_stock_shares\": num_stock_shares,\n",
    "    \"buy_cost_pct\": buy_cost_list,\n",
    "    \"sell_cost_pct\": sell_cost_list,\n",
    "    \"state_space\": state_space,\n",
    "    \"stock_dim\": stock_dimension,\n",
    "    \"tech_indicator_list\": INDICATORS,\n",
    "    \"action_space\": stock_dimension,\n",
    "    \"reward_scaling\": 1e-4\n",
    "}\n",
    "\n",
    "\n",
    "e_train_gym = StockTradingEnv(df = train, **env_kwargs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64EoqOrQjiVf"
   },
   "source": [
    "## Environment for Training\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xwSvvPjutpqS",
    "outputId": "3645668e-ba1f-4610-9789-bc212eb9b776"
   },
   "outputs": [],
   "source": [
    "env_train, _ = e_train_gym.get_sb_env()\n",
    "print(type(env_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HMNR5nHjh1iz"
   },
   "source": [
    "<a id='5'></a>\n",
    "# Part 6: Train DRL Agents\n",
    "* The DRL algorithms are from **Stable Baselines 3**. Users are also encouraged to try **ElegantRL** and **Ray RLlib**.\n",
    "* FinRL includes fine-tuned standard DRL algorithms, such as DQN, DDPG, Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to\n",
    "design their own DRL algorithms by adapting these DRL algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "364PsqckttcQ"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "\n",
    "if_using_a2c = False\n",
    "if_using_ddpg = False\n",
    "if_using_ppo = False\n",
    "if_using_td3 = False\n",
    "if_using_sac = True\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDmqOyF9h1iz"
   },
   "source": [
    "### Agent Training: 5 algorithms (A2C, DDPG, PPO, TD3, SAC)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uijiWgkuh1jB"
   },
   "source": [
    "### Agent 1: A2C\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GUCnkn-HIbmj",
    "outputId": "a2bdd15c-c366-4f89-e6de-6b0572c2ad23"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "model_a2c = agent.get_model(\"a2c\")\n",
    "\n",
    "if if_using_a2c:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/a2c'\n",
    "  new_logger_a2c = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_a2c.set_logger(new_logger_a2c)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0GVpkWGqH4-D"
   },
   "outputs": [],
   "source": [
    "trained_a2c = agent.train_model(model=model_a2c, \n",
    "                             tb_log_name='a2c',\n",
    "                             total_timesteps=50000) if if_using_a2c else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MRiOtrywfAo1"
   },
   "source": [
    "### Agent 2: DDPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "M2YadjfnLwgt",
    "outputId": "211f6ae4-c91e-41df-8906-df246e090f75"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "model_ddpg = agent.get_model(\"ddpg\")\n",
    "\n",
    "if if_using_ddpg:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/ddpg'\n",
    "  new_logger_ddpg = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_ddpg.set_logger(new_logger_ddpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tCDa78rqfO_a"
   },
   "outputs": [],
   "source": [
    "trained_ddpg = agent.train_model(model=model_ddpg, \n",
    "                             tb_log_name='ddpg',\n",
    "                             total_timesteps=50000) if if_using_ddpg else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_gDkU-j-fCmZ"
   },
   "source": [
    "### Agent 3: PPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "y5D5PFUhMzSV",
    "outputId": "3405d353-0a08-4855-ca59-98098968dd11"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "PPO_PARAMS = {\n",
    "    \"n_steps\": 2048,\n",
    "    \"ent_coef\": 0.01,\n",
    "    \"learning_rate\": 0.00025,\n",
    "    \"batch_size\": 128,\n",
    "}\n",
    "model_ppo = agent.get_model(\"ppo\",model_kwargs = PPO_PARAMS)\n",
    "\n",
    "if if_using_ppo:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/ppo'\n",
    "  new_logger_ppo = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_ppo.set_logger(new_logger_ppo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Gt8eIQKYM4G3"
   },
   "outputs": [],
   "source": [
    "trained_ppo = agent.train_model(model=model_ppo, \n",
    "                             tb_log_name='ppo',\n",
    "                             total_timesteps=50000) if if_using_ppo else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3Zpv4S0-fDBv"
   },
   "source": [
    "### Agent 4: TD3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JSAHhV4Xc-bh",
    "outputId": "802fef04-8df5-4df2-f710-e5b08941842f"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "TD3_PARAMS = {\"batch_size\": 100, \n",
    "              \"buffer_size\": 1000000, \n",
    "              \"learning_rate\": 0.001}\n",
    "\n",
    "model_td3 = agent.get_model(\"td3\",model_kwargs = TD3_PARAMS)\n",
    "\n",
    "if if_using_td3:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/td3'\n",
    "  new_logger_td3 = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_td3.set_logger(new_logger_td3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OSRxNYAxdKpU"
   },
   "outputs": [],
   "source": [
    "trained_td3 = agent.train_model(model=model_td3, \n",
    "                             tb_log_name='td3',\n",
    "                             total_timesteps=30000) if if_using_td3 else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dr49PotrfG01"
   },
   "source": [
    "### Agent 5: SAC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xwOhVjqRkCdM",
    "outputId": "3c848ff6-587e-43d1-b50d-a7d521a14b4f"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "SAC_PARAMS = {\n",
    "    \"batch_size\": 128,\n",
    "    \"buffer_size\": 100000,\n",
    "    \"learning_rate\": 0.0001,\n",
    "    \"learning_starts\": 100,\n",
    "    \"ent_coef\": \"auto_0.1\",\n",
    "}\n",
    "\n",
    "model_sac = agent.get_model(\"sac\",model_kwargs = SAC_PARAMS)\n",
    "\n",
    "if if_using_sac:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/sac'\n",
    "  new_logger_sac = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_sac.set_logger(new_logger_sac)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "K8RSdKCckJyH",
    "outputId": "4d65a11b-32b0-4c3f-b0da-e1b59fc6e194"
   },
   "outputs": [],
   "source": [
    "trained_sac = agent.train_model(model=model_sac, \n",
    "                             tb_log_name='sac',\n",
    "                             total_timesteps=40000) if if_using_sac else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f2wZgkQXh1jE"
   },
   "source": [
    "## In-sample Performance\n",
    "\n",
    "Assume that the initial capital is $1,000,000."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bEv5KGC8h1jE"
   },
   "source": [
    "### Set turbulence threshold\n",
    "Set the turbulence threshold to be greater than the maximum of insample turbulence data. If current turbulence index is greater than the threshold, then we assume that the current market is volatile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "efwBi84ch1jE"
   },
   "outputs": [],
   "source": [
    "data_risk_indicator = processed_full[(processed_full.date<TRAIN_END_DATE) & (processed_full.date>=TRAIN_START_DATE)]\n",
    "insample_risk_indicator = data_risk_indicator.drop_duplicates(subset=['date'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VHZMBpSqh1jG",
    "outputId": "73edde3b-1414-4a0d-f876-6460de2ff8e1"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.vix.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BDkszkMloRWT",
    "outputId": "c5df1f2f-ed52-4107-a3af-a358767ae155"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.vix.quantile(0.996)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AL7hs7svnNWT",
    "outputId": "ab5a1015-3f83-41b7-ba65-1487f3188588"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.turbulence.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "N78hfHckoqJ9",
    "outputId": "2c8a54d6-022f-4716-dc6d-22a450f5b9ab"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.turbulence.quantile(0.996)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U5mmgQF_h1jQ"
   },
   "source": [
    "### Trading (Out-of-sample Performance)\n",
    "\n",
    "We update periodically in order to take full advantage of the data, e.g., retrain quarterly, monthly or weekly. We also tune the parameters along the way, in this notebook we use the in-sample data from 2009-01 to 2020-07 to tune the parameters once, so there is some alpha decay here as the length of trade date extends. \n",
    "\n",
    "Numerous hyperparameters – e.g. the learning rate, the total number of samples to train on – influence the learning process and are usually determined by testing some variations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cIqoV0GSI52v"
   },
   "outputs": [],
   "source": [
    "e_trade_gym = StockTradingEnv(df = trade, turbulence_threshold = 70,risk_indicator_col='vix', **env_kwargs)\n",
    "# env_trade, obs_trade = e_trade_gym.get_sb_env()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "id": "W_XNgGsBMeVw",
    "outputId": "56d2af8e-7ee7-4248-ce7a-3132625c61b1"
   },
   "outputs": [],
   "source": [
    "trade.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eLOnL5eYh1jR",
    "outputId": "d31d2209-05ef-41df-fd8f-ee081d427949"
   },
   "outputs": [],
   "source": [
    "trained_moedl = trained_sac\n",
    "df_account_value, df_actions = DRLAgent.DRL_prediction(\n",
    "    model=trained_moedl, \n",
    "    environment = e_trade_gym)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ERxw3KqLkcP4",
    "outputId": "780c7269-0139-42e0-8ad6-081fd93717bf"
   },
   "outputs": [],
   "source": [
    "df_account_value.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "2yRkNguY5yvp",
    "outputId": "13f1ace3-c343-4bee-e0c1-214b2f8a1372"
   },
   "outputs": [],
   "source": [
    "df_account_value.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 267
    },
    "id": "nFlK5hNbWVFk",
    "outputId": "a635843e-7732-47ce-8044-9c87a33de736"
   },
   "outputs": [],
   "source": [
    "df_actions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W6vvNSC6h1jZ"
   },
   "source": [
    "<a id='6'></a>\n",
    "# Part 7: Backtesting Results\n",
    "Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Lr2zX7ZxNyFQ"
   },
   "source": [
    "<a id='6.1'></a>\n",
    "## 7.1 BackTestStats\n",
    "pass in df_account_value, this information is stored in env class\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Nzkr9yv-AdV_",
    "outputId": "216032a2-5566-4dba-db83-7d7d880b1e88"
   },
   "outputs": [],
   "source": [
    "print(\"==============Get Backtest Results===========\")\n",
    "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
    "\n",
    "perf_stats_all = backtest_stats(account_value=df_account_value)\n",
    "perf_stats_all = pd.DataFrame(perf_stats_all)\n",
    "perf_stats_all.to_csv(\"./\"+RESULTS_DIR+\"/perf_stats_all_\"+now+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QkV-LB66iwhD",
    "outputId": "7231e720-1598-45ac-ff49-9f8d29f296fa"
   },
   "outputs": [],
   "source": [
    "#baseline stats\n",
    "print(\"==============Get Baseline Stats===========\")\n",
    "baseline_df = get_baseline(\n",
    "        ticker=\"^DJI\", \n",
    "        start = df_account_value.loc[0,'date'],\n",
    "        end = df_account_value.loc[len(df_account_value)-1,'date'])\n",
    "\n",
    "stats = backtest_stats(baseline_df, value_col_name = 'close')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "qg1kvfemrrQH",
    "outputId": "fc34eb83-c616-42b3-98c7-8f8318644061"
   },
   "outputs": [],
   "source": [
    "df_account_value.loc[0,'date']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "tt1bzL5OrsTa",
    "outputId": "59ed1fd0-5a77-40e2-8def-229148fec9db"
   },
   "outputs": [],
   "source": [
    "df_account_value.loc[len(df_account_value)-1,'date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9U6Suru3h1jc"
   },
   "source": [
    "<a id='6.2'></a>\n",
    "## 7.2 BackTestPlot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "lKRGftSS7pNM",
    "outputId": "db260938-69a7-4417-fed7-25679d43b0f6"
   },
   "outputs": [],
   "source": [
    "print(\"==============Compare to DJIA===========\")\n",
    "%matplotlib inline\n",
    "# S&P 500: ^GSPC\n",
    "# Dow Jones Index: ^DJI\n",
    "# NASDAQ 100: ^NDX\n",
    "backtest_plot(df_account_value, \n",
    "             baseline_ticker = '^DJI', \n",
    "             baseline_start = df_account_value.loc[0,'date'],\n",
    "             baseline_end = df_account_value.loc[len(df_account_value)-1,'date'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "_gDkU-j-fCmZ",
    "3Zpv4S0-fDBv"
   ],
   "name": "Stock_NeurIPS2018.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
